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SUMMARY 

A new  concept  has  been  evaluated  to  support  decision 
making  in  teams.  The  concept  encompasses  a shared 
representation  and  interactive  use  of  planning 
information  in  a team  environment,  and  consists  of 
individual  workplaces,  generation  and  representation  of 
ideas,  and  shared  interactive  large  screen  displays.  This 
so-called  SmartStaff  concept  has  been  evaluated  during  a 
simulated  operation  by  the  Task  Group  Staff  of  the 
Royal  Netherlands  Navy.  By  means  of  questionnaires  the 
staff  members  were  asked  to  assess  their  current  work 
environment  and  the  potentials  of  SmartStaff.  The  results 
show  that  the  concept  provides  better  general  support  for 
group  decision  making.  SmartStaff  supported  better  the 
presentation  and  conveyance  of  ideas,  facilitated  time 
management  and  decreased  the  ambiguities  of  the  plans 
presented.  However,  the  quality  of  the  final  plan  did  not 
improve. 
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1 INTRODUCTION 

A concept  for  supporting  a planning  task  in  a team  has 
been  evaluated.  The  goal  of  this  so-called  SmartStaff 
concept  is  to  support  a team  in  developing  a common 
representation  of  both  the  problem  space  (an  operational 
situation)  and  the  solution  space  (a  plan  to  be 
developed).  This  concept  has  been  implemented  in  an 
environment  where  planners  can  work  both  individually 
and  together,  while  using  and  producing  information  in  a 
highly  interactive  way. 

A number  of  researchers  employ  the  term  ‘shared 
mental  model’  in  explaining  effective  team  behaviour 
(e.g.,  Orasanu,  1990,  Rouse,  Cannon-Bowers  & Salas, 
1992,  Stout,  1995).  The  SmartStaff  concept  encompasses 
a representation  of  a shared  mental  model  of  the 
operational  situation.  SmartStaff  also  includes  a 
representation  of  a shared  mental  model  of  the  plan  that 
the  team  is  developing. 

To  improve  our  understanding  of  team  support,  our  aim 
is  not  to  evaluate  a particular  environment  but  rather  the 
concept  behind  it.  In  order  to  evaluate  the  concept 


empirically,  we  have  implemented  the  concept  in  a naval 
command  and  control  task  of  the  Task  Group  Staff  of  the 
Royal  Netherlands  Navy  (RNLN)  that  has  operational 
command  over  a number  of  naval  platforms. 

Supporting  naval  command  and  control  teams  has  been 
the  subject  of  a large  research  program  called  TADMUS 
(Tactical  Decision  Making  Under  Stress);  see  Cannon- 
Bowers  and  Salas  (1998)  for  an  overview.  Morisson, 
Kelly,  Moore  and  Hutchins  (1998)  discuss  a number  of 
decision  support  systems  developed  in  this  program,. 
The  design  of  these  systems  were  based  on  the 
naturalistic  decision  making  theory  (Zsambok  and  Klein, 
1997),  stating  that  a decision  is  most  of  the  time  based 
on  a recognition  of  a previously  experienced  pattern. 
Decision  support  should  therefore  facilitate  the 
recognition  of  these  patterns.  Example  decision  support 
systems  designed  by  Morisson  et  al.  are  a geo-plot  (a 
computer  graphic  representation  of  a geographic  area 
with  associated  information,  i.e.  land  masses,  political 
boundaries,  symbols  for  assets  and  units),  and  a track 
profile  (graphically  displaying  the  altitude  of  an  air 
contact  over  time  and  range  from  own  ship).  However, 
supporting  team  planning  has  not  been  part  of  the 
research  program. 

A new  conceptual  approach  is  an  important  aspect  of  the 
development  of  a new  command  frigate,  the  platform 
that  embarks  the  Task  Group  Staff.  Traditionally,  the 
RNLN  designs  its  own  frigates.  Since  the  start  of 
operation  of  the  current  command  frigate,  three  decades 
ago,  much  has  changed  in  the  field  of  information  and 
communication  technology.  New  ways  of  working  have 
to  be  designed,  updated  not  only  to  the  current  state  of 
technology  but  also  to  be  prepared  for  developments  in 
the  future.  Therefore,  a conceptual  evaluation  is  more 
valuable  than  merely  a state-of-the-art  based 
technological  evaluation. 

1.1  Technological  Development 

A most  significant  change  in  the  past  decades  is  the  use 
of  electronic  information  in  groups.  McGrath  and 
Hollingshead  (1994)  give  three  reasons  why  teams 
should  work  with  electronic  information: 

• it  can  improve  task  performance, 

• it  can  overcome  time  and  space  constraints,  and 

• it  enhances  information  retrieval  and  exchange. 


Paper  presented  at  the  RTO  HFM  Symposium  on  "Usability  of  Information  in  Battle  Management 
Operations”,  held  in  Oslo,  Norway,  10-13  April  2000,  and  published  in  RTO  MP-57. 
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Important  recent  developments  in  command  and  control 
are  the  paperless  ship,  the  large  screen  displays,  and 
various  electronic  support  tools.  Electronic  support  tools 
can  be  categorised  in  different  ways.  Group  support 
systems  can  be  distinguished  on  the  basis  of  time  and 
space  constraints  (DeSanctis  and  Gallupe,  1987, 
Grudin,  1997).  People  can  work  together  at  the  same 
place  or  at  different  locations,  and  also  work  together 
synchronously  or  a-synchronously.  A second  distinction 
of  group  support  systems  is  type  of  task.  Computer 
supported  cooperative  work  can  be  divided  in: 

• communication  between  co-workers; 

• creating  and  maintaining  a shared  information 
space;  and 

• coordination  of  the  various  interactions  between  the 
co-workers,  and  between  a worker  and  the 
information  system. 

Group  systems  that  support  communication  mainly  deal 
with  groups  that  are  distributed  in  space,  some  of  which 
work  synchronously  (telephone,  video  conferencing), 
and  others  a-synchronously  (e  g.,  e-mail).  Desktop 
conferencing  is  an  example,  given  by  Gmdin  (1997),  of 
a type  of  support  system  that  creates  and  maintains  a 
shared  information  environment  in  which  group 
members  can  share  large  screen  displays  and  different 
electronic  tools.  There  are  several  group  decision  support 
system,  for  example  computer  conferencing  tools, 
application  sharing  systems,  collaborative  virtual 
environments,  audio  conferencing  systems,  and 
collaborative  software  engineering  systems  (Gmdin, 
1997,  Ter  Hofte,  1998). 

A rather  new  device  that  can  support  group  work  is 
the  shared  electronic  whiteboard.  Originally,  shared 
whiteboards  were  used  for  groups  working  at  different 
locations  who  need  to  work  on  a common  object  (e.g.,  a 
document)  during  video  conferencing,  but  it  has  been 
found  to  be  useful  in  face-to-face  meetings  too.  Streitz, 
Geissler,  Haake  & Hoi  (1994),  for  example,  compared 
three  conferencing  configurations.  In  one  configuration, 
a group  of  graphical  designers  were  provided  with 
individual  workstations.  In  a second  configuration,  an 
interactive  whiteboard  was  provided.  The  third 
configuration  consisted  of  the  mixture  of  both.  They 
found  that  designers  supported  by  both  individual 
workstations  and  an  interactive  whiteboard  performed 
best,  in  terms  of  quantity  and  quality  of  ideas,  amount  of 
activity,  and  a shared  picture  of  the  subjects  of 
discussion.  The  whiteboard  that  was  used  presented  a 
computer  screen  and  enabled  direct  interaction  or 
interaction  from  behind  the  individual  workstations.  The 
whiteboard  also  allows  drawing  pictures  with  an 
electronic  pen.  It  appears  that  this  concept  improves 
performance,  because  the  whiteboard  focuses  attention 
oil  the  design  object  as  well  as  the  design  process,  and 
facilitates  the  comparisons  of  ideas  (plans);  while  idea 
generation  takes  place  interactively  at  the  whiteboard, 
other  members  can  respond  to  it  immediately. 


1.2  Support  Concept 

Planning  may  be  defined  as  designing  a sequence  of 
actions  to  be  taken  in  order  to  react  upon  an  anticipated 
threat  with  regard  to  a mission,  where  all  actions  are 
heavily  interdependent  and  where  design  decisions  need 
confirmed  arguments.  Team  planning  requires  that 
information  needed  for  planning  as  well  as  the  plan  itself 
is  shared  and  that,  for  efficiency  reasons,  team  members 
can  work  both  collectively  and  individually.  It  is 
essential  that  the  separate,  diverging  work  of  members 
having  individual  expertise  is  followed  by  convergence 
of  ideas  in  the  team. 

Current  electronic  tools  do  not  support  team  planning. 
With  an  electronic  conference  tool,  for  instance,  a group 
can  first  generate  a list  of  individual  ideas  and  next  come 
to  an  agreement  of  the  best  one.  For  staff  planning, 
however,  a tool  is  needed  that  enables  staff  members  to 
develop  a single  plan  or  ideas  for  plan  refinement  in  a 
collective  and  integrated  way  rather  than  enabling  only 
individual  disconnected  idea  generation.  Further,  we 
think  that  for  interactive  planning,  it  is  not  only 
important  to  share  planning  information  and  the  plan 
itself;  sharing  information  about  the  planning  process  is 
essential  as  well.  Information  about  the  planning  process 
gives  other  members  the  possibility  of  reacting 
immediately  during  planning.  They  may  contribute 
concurrently  instead  of  sequentially  (i.e.,  first  developing 
a partial  plan  individually,  and  then  discussing  it  within 
the  team.) 

On  the  basis  of  these  ideas  the  SmartStaff  was 
conceptualized,  having  the  following  characteristics.  A 
planning  staff  needs: 

1 . Both  individual  and  shared  workspaces 

2.  Flow  of  information 

3.  A common  focus  of  attention 

4.  Concurrent  idea  generation 

An  “idea”  is  used  as  the  unit  for  information 
conveyance.  An  idea  can  take  various  forms:  text,  an 
object  (graphical,  but  potentially  also  audio),  or  even  a 
reference  link  to  another  idea.  Ideas  can  consists  of  sub- 
ideas. A plan  is  typically  a compound  idea. 

We  have  implemented  this  concept  with  eight  individual 
workstations  and  two  large  interactive  touch  screen 
displays  (electronic  whiteboards),  electronic  storage, 
retrieval  and  exchange  of  information,  and  an  electronic 
idea  pad:  a tool  for  both  individual  and  team  generation 
and  representation  of  ideas.  To  guarantee  face-to-face 
contact  and  an  unlimited  view  of  the  large  screen 
displays,  the  workstations  were  lowered  and  positioned 
in  a semi-circle  around  the  large  screen  displays.  Figure 
1 shows  the  layout  of  the  experimental  staff  room. 

As  idea  pad  we  used  a commercially  available  software 
tool  (SmartNotebook,  from  Smart  Technologies  Inc.), 
but  in  a specific  way.  In  the  idea  pad,  ideas  may  be  put 
on  a single  page,  or  divided  over  different  pages.  The 
development  of  an  idea  may  be  recorded,  enabling 
skipping  back  to  an  earlier  development  phase.  Elements 
of  an  idea  can  be  made  from  scratch  or  may  be  imported 
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from  existing  sources,  including  other  idea  pads  or  idea 
pad  pages.  Other  available  electronic  information 
sources  are  the  tactical  situation,  meteorological 
information,  charts,  mission  statement,  messages, 
intelligence,  etc. 


Figure  1 : The  layout  of  the  experimental  staff  room, 


Ideas  can  be  exchanged  with  a specially  developed  tool. 
When  a team  member  wants  to  submit  an  idea  he  can 
send  his  idea  pad  to  a common  idea  repository,  presented 
at  the  large  screen  display  with  the  name  of  the  sender, 
and  a one-line  description.  The  content  of  the  private 
idea  pad  can  be  discussed  within  the  team,  and  accepted 
(possibly  after  revision  by  the  individual  or  the  team)  as 
(part  of)  a shared  team  idea,  or  thrown  away.  The  other 


way  around,  a shared  team  idea  can  be  fetched  from  the 
common  idea  repository  and  placed  on  the  individual’s 
own  work  space,  for  example  for  extension,  correction, 
or  refinement. 

1.3  Task  Analysis 

This  paper  describes  the  empirical  evaluation  of  the 
SmartStaff  concept  in  the  naval  command  and  control 
task  of  task  group  staff  planning.  The  aim  of  this  study  is 
to  examine  whether  the  team  decision  making 
performance  improves  when  the  team  is  supported  by  the 
SmartStaff  concept.  More  specifically,  we  want  to 
investigate  whether  SmartStaff  improves  a shared  picture 
of  the  situation  and  the  plan,  is  more  time-efficient, 
improves  the  communication  of  ideas  and  the  quality  of 
the  final  plan. 

Before  the  evaluation  we  first  analysed  the  work  of  the 
Staff.  Naval  platforms  seldom  operate  individually,  but 
rather  in  a group,  called  a Task  Group.  The  Task  Group 
Commander,  supported  by  a team  varying  from  five  to 
fifteen  members  with  a specific  individual  expertise, 
such  as  in  meteorology,  intelligence,  communications, 
and  the  different  warfare  areas,  exercises  command  from 
a dedicated  frigate.  Monitoring,  threat  assessment,  and 
control  of  operations  of  the  whole  Task  Group  takes 
place  in  the  staff  room  of  this  command  frigate.  Various 
phases  of  an  operational  situation  can  be  distinguished, 
differing  in  threat,  workload  and  intensity  of  the  team 
decision  making  process.  The  planning  process  is 
illustrated  in  figure  2. 


Figure  2:  Staf  tasks.  The  rounded  boxes  are  tasks;  the  shaded  boxes  are  information  boxes;  arrows  are  data 
dependencies 
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Based  on  its  mission,  the  Staff  plans  the  activities  of  the 
task  group  and  executes  the  plan  by  directing  the  units. 
During  execution,  both  the  situation  and  the  execution 
progress  is  monitored.  When  the  situation  changes  and 
problems  in  executing  the  plan  are  anticipated,  or  the 
mission  changes,  the  plan  is  revised.  This  process  is 
carried  out  in  three  cycles: 

1.  long-term  (re)planning  (more  than  half  a day  in 
advance) 

2.  short-term  (re)planning  (up  to  half  a day  in  advance) 

3.  near-real-time  decision  making 

During  these  cycles,  briefings  take  place  regularly  to 
inform  the  members  of  the  staff.  Re-planning  occurs  by 
generating  collectively  a solution  in  a rough  form,  and 
then  working  out  the  details  in  the  plan  individually,  and 
next,  discussing  them  collectively.  The  final  adaptations 
are  translated  into  orders  and  sent  to  the  task  group  units. 

In  its  current  environment,  the  Task  Group  Staff  does 
not  work  much  with  electronic  information.  The  Staff 
does  not  have  electronic  presentation  or  electronic  data 
exchange  facilities;  much  is  done  on  paper,  and  on  white 
boards  and  tote  boards. 

2 METHOD 

2.1  Subjects 

All  14  RNLN  Task  Group  Staff  members  served  as 
subjects,  6 Petty  Officers  1st  class,  6 1st  Lieutenants,  a 
Captain  and  a Commodore.  The  1st  Lieutenants  were 
experts  in  one  or  more  particular  areas  (operations,  the 
three  warfare  areas,  meteorology,  communication,  and 
intelligence).  The  Captain  and  the  Commodore  were  the 
team  leaders.  The  Petty  Officers  supported  the  Officers. 
All  subjects  had  significant  operational  experience,  also 
within  this  team  (except  for  the  Captain).  Mean  age  was 
39  years  (34  to  50). 

2.2  Design 

The  RNLN  Task  Group  Staff  carried  out  their  work 
during  a simulated  operation  in  the  SmartStaff-based 
environment.  For  pragmatic  reasons,  we  were  not  able  to 
make  a pure  experimental  comparison  between  this 
environment  and  an  environment  not  based  on 
SmartStaff  . Only  one  RNLN  Task  Group  Staff  exists 
and  its  time  is  restricted.  Therefore,  the  staff  members 
were  asked  to  compare  this  experimental  environment 
with  their  normal  working  environment.  To  make  their 
work  as  similar  as  possible,  we  used  an  operational 
scenario  that  was  comparable  with  a training  scenario 
they  had  used  earlier  in  their  current  environment.  In 
addition  to  this  self  assessment,  we  invited  two  experts 
in  the  field  as  independent  observers,  to  collect 
information  for  the  interpretations  of  the  results. 

Three  questionnaires  were  developed:  one  for 
assessing  the  current  environment  (A),  one  for  assessing 
SmartStaff  (B),  and  one  for  comparing  directly  both 
environments  (C).  With  the  three  questionnaires, 


SmartStaff  was  tested  in  two  ways: 

• Indirectly,  by  comparing  the  questions  about  the 
current  environment  (questionnaire  A)  with  the 
questions  about  SmartStaff  (questionnaire  B) 

• Directly,  by  testing  the  null  hypothesis  that 
SmartStaff  and  the  current  environment  supports  the 
Task  Group  Staff  equally  well.  This  is  done  by 
comparing  the  questions  in  which  both 
environments  were  compared  (questionnaire  C) 
with  the  answer  ‘equal’. 

The  subjects  as  well  as  the  observers  filled  out  the  three 
questionnaires.  The  questions  in  each  questionnaire  were 
organised  in  5 modules: 

1.  point  of  focus  and  shared  picture  (3  questions),  e.g., 
“How  often  during  a meeting  in  <the  current 
environment  do  you  have  a different  picture  of  the 
situation  than  a colleague?” 

2.  efficient  use  of  individual  and  shared  time  (5 
questions),  e.g.,  “How  often  during  <the  SmartStaff 
meeting>  do  you  experience  that  you  lose  time?” 

3.  communication  of  ideas  (5  questions),  e.g.,  “Can 
you  clarify  your  ideas  with  SmartStaff  better  than  in 
the  current  environment?. 

4.  product  quality  (1  question),  e.g.,  “How  do  you 
assess  the  mean  quality  of  the  plans  resulting  from 
the  <current  environment”? 

5.  general  questions  (12  questions)  , e.g.,  “How  well 
can  you  present  your  ideas  to  the  team  in  <the 
current  environment?”;  “How  well  does 
<SmartStaff>  support  you  in  participating  in  the 
team  discussion?”. 

In  questionnaire  A and  B,  answers  had  to  be  given  on  a 
4-point  scale  (bad,  rather  bad,  rather  good,  good).  In 
questionnaire  C,  a 5-point  Likert  scale  was  used  (with 
answer  categories  ‘much  worse’,  ‘worse’,  ‘equal’, 
‘better’,  ‘much  better’).  This  questionnaire  also  asked 
some  open  question  about  both  environments,  such  as 
about  their  strengths  and  weaknesses,  possible 
improvements,  and  the  potential  of  SmartStaff,  e.g.,  “Do 
you  have  any  suggestion  for  improving  SmartStaff?” 

The  subjects  as  well  as  the  observers  also  took  part  in  a 
group  discussion,  taking  place  after  having  experienced 
the  SmartStaff  environment.  The  group  discussion  was 
also  based  on  qualitative  questions. 

2.3  Scenarios 

A realistic  simulation  in  the  environment  described 
above  requires  a full  scenario  in  which  the  Staff  directs  a 
Task  Group  consisting  of  various  frigates,  tankers,  an 
amphibious  unit,  air  units,  and  a submarine,  within 
political  constraints  laid  down  in  so  called  Rules  of 
Engagement.  The  scenario  was  based  on  a training 
scenario,  adapted  to  the  above  mentioned  three  team 
decision  making  cycles. 

For  long-term  planning,  the  task  was  to  prepare  a plan  to 
escort  a Task  Unit  to  a particular  waiting  area  prior  to  an 
amphibious  landing  by  NATO  forces,  and  to  execute  the 
plan  within  76  hours.  For  short-term  planning  the  task 
was  to  formulate  a group  assessment  of  the  present 
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tactical  situation,  including  a tentative  identification  of 
surface  contacts,  and  to  (re)task  available  assets  and 
units  in  order  to  accomplish  the  task.  For  near-real-time 
decision  making,  the  Staff  had  to  assess  the  development 
of  the  tactical  situation,  to  reconsider  eventually  the 
identity  of  the  surface  contacts,  and  to  decide  on 
manoeuvring  the  formation  or  to  engage.  Information 
needed  for  planning  and  decision  making,  such  as 
mission,  mles  of  engagement,  observations, 
meteorology,  intelligence,  possible  threats,  etc.,  was 
made  available  electronically. 

2.4  Procedure 

One  week  in  advance,  the  information  used  in  the 
scenario,  consisting  of  30  pages  of  text,  sea  charts,  etc., 
was  provided  to  the  Task  Group  Staff.  Data  collection 
took  place  in  one  afternoon,  from  12  to  6 pm.  After  a 
short  explanation  of  the  aim  of  the  study,  the  subjects 
filled  out  questionnaire  A.  After  lunch,  the  SmartStaff 
concept  was  introduced  and  explained.  Next,  the  subject 
were  trained  for  one  hour  in  using  the  support  tools, 
working  through  a number  of  exercises  about  forming 
and  presenting  ideas,  and  sending  and  fetching  them. 
The  game  started  at  2.30  and  lasted  for  two  hours.  In  the 
first  hour  of  the  game,  long-term  planning  took  place, 
without  the  Task  Group  Commander  (TGC).  The  plan 
was  subsequently  briefed  to  the  TGC.  In  the  remaining 
time,  short-term  planning  and  near-real-time  decision 
making  was  carried  out,  as  a reaction  to  a developing 
threat.  In  these  tasks,  the  TGC  participated  fully.  After  a 
break,  questionnaires  B and  C were  filled  in.  The  session 
concluded  with  the  group  discussion. 

2.5  Results 

The  results  came  from  the  three  questionnaires  filled  in 
by  eight  staff  officers,  the  questionnaire  filled  in  by  one 


observer  (the  second  observer  didn’t  show  up),  and 
remarks  made  during  the  group  discussion.  Reliability 
analysis  showed  that  two  items  in  module  2 (efficient 
use  of  individual  and  shared  time)  had  low  item-total 
correlation;  these  items  were  left  out  of  the  analysis.  The 
reliability  of  the  remaining  questionnaires  was 
reasonable  to  good  (Crombach’s  alpha  .97  for 
questionnaire  A;  .74  for  B;  and  .71  for  C). 

In  table  1 the  results  are  presented  for  the  four  specific 
performance  criteria,  derived  from  the  first  four  modules 
from  the  questionnaires.  A fifth  overall  performance 
assessment  is  added,  derived  from  all  five  modules.  A 
Wilcoxon  rank  test  was  used.  Each  performance  criterion 
was  tested  by  averaging  the  scores  across  the  questions 
of  a module.  The  table  shows  the  mean  scores  together 
with  their  standard  deviation  (between  brackets),  and  the 
level  of  significance  (p-values). 

The  Task  Group  Staff  assessed  that  the  SmartStaff  based 
environment  supported  their  decision  making  better  than 
their  current  environment  (p  < 0.05  for  both 

comparisons).  Important  to  remark  is  that  the  subjects 
regularly  noted  they  evaluated  in  questionnaires  B and  C 
the  potential  of  the  SmartStaff  concept,  not  the  current 
experimental  implementation,  for  which  is  clear  that 
certain  interaction  mechanisms  and  the  speed  of  data 
exchange  can  be  improved. 

SmartStaff  was  found  to  provide  a less  ambiguous 
shared  picture  shared  of  the  situation  and  the  plan,  when 
the  two  environment  were  compared  directly  (p  < 0.05). 
No  significant  difference  was  found  for  the  absolute 
assessments  of  the  two  environments. 

The  Staff  also  had  the  opinion  that  with  SmartStaff  their 
time  was  used  more  efficiently.  Again,  this  result  was 
only  found  in  the  direct  comparison  (p  < 0.5).  A 
drawback,  put  forward  by  some  subjects,  was  that 
carrying  out  individual  work  in  the  SmartStaff 
environment  may  distract  one  from  shared  decision 
making. 


Performance  criterion 

Mean  Score  (Standard  deviation) 

p-values  of  difference 

A 

B 

C 

A-B 

C-‘equal’ 

Cr.a=.  79 

Cr.a=.74 

Cr.a=,71 

overall  performance 
(module  1 to  5;  24 

2.81  (0.32) 

3.01  (0.30) 

3.77  (0.25) 

.05 

.01 

questions) 

shared  situation  picture 
(module  1 ; 3 questions) 

2.95  (0.49) 

3.29  (0.42) 

3.49  (0.32) 

n.s. 

.02 

time  efficiency 
(module  2;  3 questions) 

2.96  (0.49) 

2.92  (0.43) 

3.65  (0.49) 

n.s. 

.02 

idea  communication 

2.66  (0.32) 

3.08  (0.55) 

4.22  (0.41) 

.06 

.01 

(module  3;  5 question) 
product  quality 
(module  4;  1 question) 

3.25  (0.46) 

3.13  (0.35) 

3.33  (0.41) 

n.s. 

n.s. 

Cr.a:  Crombach’s  alpha;  n.s.:  not  significant; 

A:  Current  environment;  values  ranging  from  1-4  (bad,  rather  bad,  rather  good,  good); 

B:  SmartStaff  environment;  values  ranging  from  1-4  (bad,  rather  bad,  rather  good,  good); 

C:  Direct  comparison;  values  ranging  from  1-5  (much  worse,  worse,  equal,  better,  much  better). 


Table  1 : Overview  of  the  results 
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With  SmartStaff,  the  quality  of  ideas  generated  is  not 
better  compared  to  their  current  environment  (p=  0.35 
for  comparison  afterwards,  and  p-0. 1 1 for  direct 
comparison). 

The  Task  Group  Staff  was  clear  about  the  value  of 
SmartStaff  for  the  communication  of  ideas.  The  subjects 
unanimously  thought  that  the  SmartStaff  based 
environment  is  better  or  much  better  in  this  respect 
compared  to  the  current  environment.  (p=0.06  when 
indirectly  compared;  0.01  for  direct  comparison).  Also, 
in  their  comments  the  subjects  expressed  the  strength  of 
SmartStaff  on  this  aspect. 

The  assessment  of  the  observer  was  in  accordance  with 
the  assessment  of  the  Task  Group  Staff . 

4 GENERAL  DISCUSSION  AND  CONCLUSIONS 

4.1  The  results 

The  results  have  shown,  that  the  SmartStaff  concept  has 
a large  potential  to  support  the  planning  in  teams.  In 
several  respects,  a SmartStaff  based  environment 
provides  better  support  for  team  decision  making  than 
traditional  environments: 

• the  shared  picture  of  the  situation  and  plan  that  staff 
members  have  is  better  and  less  ambiguous, 

• time  is  managed  more  efficiently, 

• presenting  and  communicating  ideas  and  plans  runs 
much  better, 

• general  support  to  team  decision  making  is  better. 

In  one  respect,  SmartStaff  did  not  have  any  effect:  the 
quality  of  the  ideas  and  plans  were  not  influenced. 

The  latter  result  seems  to  be  in  contrast  with  the 
findings  of  Streitz  and  colleagues  (Streitz,  Geissler, 
Haake  and  Hoi,  1994,  Streitz,  Rcxroth,  P.  and  Holmer, 
1997).  They  found  an  improved  output  in  terms  of  both 
quality  and  quantity  when  a group  designed  a logo 
together  while  supported  by  individual  workstations  and 
shared  interactive  large  screen  displays.  In  that  task, 
however,  the  members  of  the  team  all  had  the  same 
expertise  and  did  not  work  together  on  a single  logo; 
they  generated  a number  of  them  and  then  selected  the 
best  one.  So,  the  type  of  task  and  the  homogeneity  of  the 
staff  and  the  logo  designers  are  different  and  a 
comparison  can  not  be  made  easily. 

It  would  be  interesting  to  find  out  why  the  quality  of 
the  final  plan  did  not  improve.  One  explanation  may  be 
that  in  a planning  task,  in  contrast  to  real-time  decision 
making,  sufficient  time  is  available  to  make  the  plan 
better.  A difference  in  output  would  only  be  found  when 
there  are  time  constraints.  In  such  a situation,  more 
efficient  time  management  will  play  a critical  role. 

Individual  comments  of  the  subjects  and  remarks 
during  the  group  discussion  pointed  out  a number  of 
additional  aspects.  The  advantage  of  interactive  large 
screen  displays  to  support  idea  presentation  and 
communication  was  particularly  clear  for  long-term 
planning  and  for  briefing.  A better  and  less  ambiguous 
shared  picture  was  also  found  for  short-term  planning 
and  near-real-time  decision  making.  Another  positively 


assessed  aspect  was  the  possibility  of  reviewing  the 
course  of  the  decision  making  process.  Time 
management  was  found  to  be  more  efficient:  the  subjects 
found  it  easy  to  change  from  individual  tasks  to 
collective  tasks.  It  was  recognised,  however,  that 
individual  tasks  that  demand  much  concentration,  could 
better  be  carried  out  in  isolation. 

4.2  Limitations  of  the  method 

Firm  conclusions  are  limited  by  the  method  used. 

One  methodological  problem  may  be  the  Hawthorne 
effect:  is  the  effect  not  just  a result  of  running  an 
innovative  system?  This  may  be  avoided  by  asking  the 
staff  to  work  in  new  environment  for  some  time  and 
measure  again.  Unfortunately,  this  is  practically 
unfeasible.  The  results  show,  however,  that  the  subjects 
were  not  positive  on  one  particular  aspect:  quality  and 
quantity  of  the  ideas  did  not  improve.  Also,  during  the 
group  discussion,  it  became  clear  that  there  was  at  least 
some  reluctance  to  accept  the  new  technology.  These 
two  observations  may  indicate  that  the  subjects  may  not 
have  been  influenced  by  the  Hawthorne  effect. 

Using  a real  team  instead  of  artificial  teams  has  the 
advantage  of  ecological  validity.  We  have  studied  a team 
in  its  real  working  environment,  with  members  having 
specific  individual  knowledge,  experience,  and  skills 
who  are  used  to  working  with  each  other.  A drawback  is 
the  practical  consequences.  It  is  difficult  to  carry  out 
tests  on  a Task  Group  Staff  since  there  is  only  one  such 
team  in  the  RNLN.  Still,  the  impact  of  their  decision 
making  process  is  enormous,  so  research  is  important, 
even  within  these  methodological  constraints. 

4.3  Conclusions 

Our  aim  was  not  to  fine  tuning  a particular 
implementation  but  to  carry  out  a conceptual  evaluation 
in  an  early  phase  in  a systems  engineering  life  cycle. 
Such  an  evaluation  yields  the  functional  requirements  of 
a system  (see  e.g.  Sage,  1992).  Neerincx,  Van  Doorne 
and  Ruijsendaal  (1999)  show  also  that  support  systems 
can  indeed  be  evaluated  in  an  early  phase.  They 
distinguish  a task  level  and  a communication  level 
evaluation,  the  former  can  be  carried  out  far  before  the 
system  is  operational.  The  lessons  of  a conceptual 
evaluation  are  independent  of  the  state  of  technology, 
and  therefore  last  longer.  Moreover,  it  helps  us  to 
understand  how  teams  work  and  how  they  should  be 
supported. 

The  present  findings  raise  a lot  of  questions.  Further 
work  will  address  why  the  final  product  of  the  staff  did 
not  apparently  improve.  We  would  like  to  know  whether 
this  depends  on  group  characteristics  (a  heterogeneous 
team  used  to  work  together)  or  type  of  task  (no  time 
contstraints).  A second  question,  following  from  the 
first  one,  is  whether  other  types  of  teams  may  profit  from 
SmartStaff  also.  Does  a management  board  experience 
the  same  level  of  support?  A third  question  is  how  team 
planning  and  refinement  is  carried  out  at  a cognitive 


5-7 


level.  For  complex  tasks,  a human  being  has  a number  of 
mechanisms  available  for  problem  solving 
simplification,  such  as  satisficing  (Simon,  1978). 
Satisficing  is  a problem  solving  strategy  often  used  by 
designers:  they  are  satisfied  with  a solution  to  their 
problem  when  it  satisfies  the  constraints,  without  further 
searching  for  a better  solution.  It  may  be  the  case  that  the 
product  of  planning  and  refinement  can  only  improve 
when  the  planning  problem  or  the  problem  solving 
strategy  is  more  complex.  If  so,  team  planning  needs 
specific  cognitive  support  to  manage  the  complexity. 
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